Holistically Evaluating Agent Based Social System Models

The philosophical perspectives on model evaluation can be broadly classified into reductionist/logical positivist and relativist/holistic. In this paper, we outline some of our past efforts in, and challenges faced during, evaluating models of social systems with cognitively detailed agents. Owing to richness in the model, we argue that the holistic approach and consequent continuous improvement are essential to evaluating complex social system models such as these. A social system built primarily of cognitively detailed agents can provide multiple levels of correspondence, both at observable and abstract aggregated levels. Such a system can also pose several challenges, including large feature spaces, issues in information elicitation with database, experts and news feeds, counterfactuals, fragmented theoretical base, and limited funding for validation. We subscribe to the view that no model can faithfully represent reality, but detailed, descriptive models are useful in learning about the system and bringing about a qualitative jump in understanding of the system it attempts to model – provided they are properly validated. Our own approach to model evaluation is to consider the entire life cycle and assess the validity under two broad dimensions of (1) internally focused validity/quality achieved through structural, methodological, and ontological evaluations; and (2) external validity consisting of micro validity, macro validity, and qualitative, causal and narrative validity. In this paper, we also elaborate on selected validation techniques that we have employed in the past. We recommend a triangulation of multiple validation techniques, including methodological soundness, qualitative validation techniques, such as face validation by experts and narrative validation, and formal validation tests, including correspondence testing

Validating Agent Based Social Systems Models

Validating social systems is not a trivial task. The paper outlines some of our past efforts in validating models of social systems with cognitively detailed agents. It also presents some of the challenges faced by us. A social system built primarily of cognitively detailed agents can provide multiple levels of correspondence, both at observable and abstract aggregated levels. Such a system can also pose several challenges including large feature spaces, issues in information elicitation with database, experts and news feeds, counterfactuals, fragmented theoretical base, and limited funding for validation. Our own approach to validity assessment is to consider the entire life cycle and assess the validity under four broad dimensions of methodological validity, internal validity, external validity and qualitative, causal and narrative validity. In the past, we have employed a triangulation of multiple validation techniques, including face validation as well as formal validation tests including correspondence testing

Modeling the Personality & Cognition of Leaders

This paper summarizes efforts at adapting a personality profiling framework to model behavior and choices of political and military leaders. This is part of a larger project to create a role-playing, decision-making game to allow you to play out scenarios of interest against other leaders. In this modeling exercise we implement the Hermann leader personality profile tool to create historic leaders (Saladin, Richard I, etc.). We then attempt to validate the leader agents against scenarios of the 3rd Crusade

Profiling is Politically \u27Correct\u27: Agent-Based Modeling of Ethno-Political Conflict

A holy grail for military, diplomatic, and intelligence analysis is a valid set of software agent models that act as the desired ethno-political factions so that one can test the effects that may arise from alternative courses of action in different lands. This article enumerates the challenges of such a testbed and describes best-of-breed leader and follower profiling models implemented to improve the realism and validity of the agent. Realistic, \u27descriptive\u27 agents are contrasted to rational actor theory in terms of the different equilibria one would expect to emerge in conflict games. These predictions are examined in two real world cases (Iraq and SE Asia) where the agent models are subjected to validity tests and a policy experiment is then run. We conclude by arguing that substantial effort on game realism, best-of-breed social science models, and agent validation efforts is essential if analytic experiments are to effectively explore conflicts and alternative ways to influence outcomes. Such efforts are likely to improve behavioral game theory as well

What is a Good Pattern of Life Model? Guidance for Simulations

We have been modeling an ever-increasing scale of applications with agents that simulate the pattern of life (PoL) and real-world human behaviors in diverse regions of the world. The goal is to support sociocultural training and analysis. To measure progress, we propose the definition of a measure of goodness for such simulated agents, and review the issues and challenges associated with first-generation (1G) agents. Then we present a second generation (2G) agent hybrid approach that seeks to improve realism in terms of emergent daily activities, social awareness, and micro-decision making in simulations. We offer a PoL case study with a mix of 1G and 2G approaches that was able to replace the pucksters and avatar operators needed in large-scale immersion exercises. We conclude by observing that a 1G PoL simulation might still be best where large-scale, pre-scripted training scenarios will suffice, while the 2G approach will be important for analysis or if it is vital to learn about adaptive opponents or unexpected or emergent effects of actions. Lessons are shared about ways to blend 1G and 2G approaches to get the best of each

Challenges of Country Modeling with Databases, Newsfeeds, and Expert Surveys

According to expert practitioners and researchers in the field of human behavior modeling ([Silverman et al., 2002; Pew and Mavor, 1998; Ritter et al., 2003]), a common central challenge now confronting designers of HBM (human-behavior-modeling) applications is to increase the realism of the synthetic agents\u27 behavior and coping abilities. It is well accepted in the HBM (human-behavior-modeling) community that cognitively detailed, thick models are required to provide realism. These models require that synthetic agents be endowed with cognition and personality, physiology, and emotive components. (We will hereafter refer to these rich models as cognitively detailed models or thick agents. ) To make these models work, one must find ways to integrate scientific know-how from many disciplines, and to integrate concepts and insights from hitherto fragmented and partial models from the social sciences, particularly from psychology, cultural studies, and political science. One consequence of this kind of integration of multiple and heterogeneous concepts and models is that we frequently end up with a large feature space of parameters that then need to be filled in with data

Human Behavior Models for Agents in Simulators and Games: Part II Gamebot Engineering with PMFserv

Many producers and consumers of legacy training simulator and game environments are beginning to envision a new era where psycho-socio-physiologic models could be interoperated to enhance their environments\u27 simulation of human agents. This paper explores whether we could embed our behavior modeling framework (described in the companion paper, Part 1) behind a legacy first person shooter 3D game environment to recreate portions of the Black Hawk Down scenario. Section 1 amplifies the interoperability needs and challenges confronting the field, presents the questions that are examined, and describes the test scenario. Sections 2 and 3 review the software and knowledge engineering methodology, respectively, needed to create the system and populate it with bots. Results (Section 4) and discussion (Section 5) reveal that we were able to generate plausible and adaptive recreations of Somalian crowds, militia, women acting as shields, suicide bombers, and more. Also, there are specific lessons learned about ways to advance the field so that such interoperabilities will become more affordable and widespread

Modeling Factions for \u27Effects Based Operations\u27: Part II Behavioral Game Theory

Military, diplomatic, and intelligence analysts are increasingly interested in having a valid system of models that span the social sciences and interoperate so that one can determine the effects that may arise from alternative operations (courses of action) in different lands. Part I of this article concentrated on internal validity of the components of such a synthetic framework – a world diplomacy game as well as the agent architecture for modeling leaders and followers in different conflicts. But how valid are such model collections once they are integrated together and used out-of-sample (see Section 1)? Section 2 compares these realistic, descriptive agents to normative rational actor theory and offers equilibria insights for conflict games. Sections 3 and 4 offer two real world cases (Iraq and SE Asia) where the agent models are subjected to validity tests and an EBO experiment is then run for each case. We conclude by arguing that substantial effort on game realism, best-of-breed social science models, and agent validation efforts is essential if analytic experiments are to effectively explore conflicts and alternative ways to influence outcomes. Such efforts are likely to improve behavioral game theory as well

Modeling Factions for \u27Effects Based Operations\u27: Part I Leader and Follower Behaviors

This paper presents a synthetic approach for generating role playing simulation games intended to support analysts (and trainees) interested in testing alternative competing courses of action (operations) and discovering what effects they are likely to precipitate in potential ethno-political conflict situations. Simulated leaders and followers capable of playing these games are implemented in a cognitive modeling framework, called PMFserv, which covers value systems, personality and cultural factors, emotions, relationships, perception, stress/coping style and decision making. Of direct interest, as Sect. 1.1 explains, is mathematical representation and synthesis of best-of-breed behavioral science models within this framework to reduce dimensionality and to improve the realism and internal validity of the agent implementations. Sections 2 and 3 present this for leader profiling instruments and group membership decision-making, respectively. Section 4 serves as an existence proof that the framework has generated several training and analysis tools, and Sect. 5 concludes with lessons learned. Part II turns to the question of assessment of the synthesis and its usage in course of action studies

A Systems Approach to Healthcare: Agent-based Modeling, Community Mental Health, and Population Well-being

Purpose Explore whether agent-based modeling and simulation can help healthcare administrators discover interventions that increase population wellness and quality of care while, simultaneously, decreasing costs. Since important dynamics often lie in the social determinants outside the health facilities that provide services, this study thus models the problem at three levels (individuals, organizations, and society). Methods The study explores the utility of translating an existing (prize winning) software for modeling complex societal systems and agent\u27s daily life activities (like a Sim City style of software), into a desired decision support system. A case study tests if the 3 levels of system modeling approach is feasible, valid, and useful. The case study involves an urban population with serious mental health and Philadelphia\u27s Medicaid population (n = 527,056), in particular. Results Section 3 explains the models using data from the case study and thereby establishes feasibility of the approach for modeling a real system. The models were trained and tuned using national epidemiologic datasets and various domain expert inputs. To avoid co-mingling of training and testing data, the simulations were then run and compared (Section 4.1) to an analysis of 250,000 Philadelphia patient hospital admissions for the year 2010 in terms of re-hospitalization rate, number of doctor visits, and days in hospital. Based on the Student t-test, deviations between simulated vs. real world outcomes are not statistically significant. Validity is thus established for the 2008–2010 timeframe. We computed models of various types of interventions that were ineffective as well as 4 categories of interventions (e.g., reduced per-nurse caseload, increased check-ins and stays, etc.) that result in improvement in well-being and cost. Conclusions The 3 level approach appears to be useful to help health administrators sort through system complexities to find effective interventions at lower costs